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Setting Standards for the 1998 NAEP in Civics and Writing: 
Using Focus Groups to Finalize the Achievement Levels Descriptions 



Patricia L. Hanick and Susan Cooper Loomis 
ACT, Inc. 



Introduction 

The description of achievement levels is critically important to the process of reporting student 
performance on the National Assessment of Educational Progress (NAEP). The National Assessment 
Governing Board (NAGB) has adopted general policy definitions of three achievement levels - B^ic, 
Proficient, and Advanced - that apply to all academic subjects at all grade levels tested by NAEP. NAGB 
has been charged by Congress to generate achievement levels that are “useful, reasonable, and valid” 
(Public Law 1 03-382, Improving America ’s Schools Act of 1994). The development of achievement levels 
descriptions (ALDs) is an important part of this mandate. 

Preliminary achievement levels descriptions were developed for writing and civics as part of the 1998 
Assessment Frameworks. The preliminary ALDs are more specific than the generic NAGB policy 
definitions in that they operationally define student performance in the content areas assessed. The 
preliminary ALDs are statements of what students should know and be able to do at each of the 
achievement levels at 4*, 8*, and 12* grades for civics and writing. 

Past Procedures Modified 

In the past, the job of developing the ALDs was one of three major tasks carried out by panelists 
participating in the achievement levels -setting (ALS) sessions. Their responsibilities included: 

1 . Determining content-based descriptions of each level of achievement for each of the three grades 
assessed by NAEP; 

2. Determining numerical cutpoints on the assessment that tied these descriptions to performance on 
NAEP; and 

3. Determining items that illustrate the skills and knowledge characterizing student performance at each 
of the three achievement levels at each of the three grades assessed by NAEP in these subjects. 

For the 1 998 NAEP, these duties were altered. Instead of ALS panelists fulfilling the first task of 
determining the ALDs for the 1998 NAEP, the achievement levels descriptions were finalized before 
convening the ALS participants. ACT designed a series of expert discussions and reviews that resulted in 
modifications and refinement of the preliminary ALDs prior to convening the ALS panels. Panelists 
engaged in the ALS process no longer were responsible for modifying and refining the ALDs. This is an 
improvement in the ALS process over the previously used method for several reasons: 

• It reduced the time required of panelists in an already full ALS schedule. 

• It decreased the cognitive demands placed on panelists during the ALS session. 

• It allowed more time to deliberate and develop the ALDs. 

• It involved more stakeholders in the process of developing the ALDs. 

• It enabled evaluation of the ALDs before they were used in the ALS process. 

Plan for Developing Achievement Levels Descriptions for the 1998 NAEP 
NAGB’ fundamental approach to setting achievement levels is based on collecting opinions fi-om a 
broadly representative sample of knowledgeable constituents who are invited to become involved in the 
ALS process. In keeping with NAGB ’s consensus-building approach to setting achievement levels for 
NAEP, ACT designed a method to conduct a series of expert reviews and discussions of the preliminary 
ALDs. 
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The process of developing the ALDs involved several steps: 

1 . Focus groups reconunended improvements to the preliminary ALDs. 

2. Expert review panels modified the ALDs after considering the focus groups’ recommendations. 

3. Opinions of the revised ALDs were collected from key individuals who had been involved in the 
development of the NAEP for civics and writing. 

4. The coverage of the content areas outlined in the Civics Framework was analyzed according to the 
revised Civics ALDs. 

5. The coverage of the Civics Item Pool was analyzed according to the revised Civics ALDs. 

The review by focus groups involved a broad segment of the population to evaluate the preliminary 
descriptions. The purpose of the evaluation was to determine whether the descriptions appeared to be both 
useful and reasonable. The judgment of “reasonableness” was with respect to the NAGB policy 
definitions of Basic, Proficient, and Advanced. The results of the focus groups’ evaluation were reviewed 
by content experts in writing and civics, who modified the ALDs accordingly. The revised ALDs were 
reviewed further by several different groups of informed stakeholders, including the original focus group 
members. The content coverage described by the revised Civics ALDs was compared with the content 
areas contained in the Civics Framework and Civics Item Pool. After extensive review and evaluation, 
ACT recommended the revised ALDs to NAGB for approval. A more detailed explanation follows of the 
plan that was implemented for developing the ALDs for the 1998 NAEP. 

Step 1: The Focus Groups 

Participants. Abt Associates Inc., imder contract to ACT, conducted focus groups in each of the four 
NAEP regions for both writing and civics. A total of eight focus groups were held. The process for 
identifying focus group participants was similar to ACT’s design for recruiting ALS panelists. Panels 
consisted of a mix of teachers, nonteacher educators, and general public members with 70-75% educators 
and 25-30% noneducators. Please see Tables 1 and 2 for the composition of the focus groups. 

In order to assure representation of important attributes among the panelists, the following guidelines 
were used in drawing the panel members; 

• Six classroom teachers (two per grade) 

• Three or four non-teacher educators (not more than two per grade) 

• One to three members of the general public (not more than one per grade) 

• Maximized variation of geographic areas within each NAEP region 

• Equal gender balance 

• Proportional ethnic and cultural diversity 

Educators were nominated by directors of assessment and testing for each state in which focus groups 
were held: Massachusetts, Georgia, Missouri, and Washington. Nominations also were submitted by 
£q}propriate staff and faculty at major teaching colleges and universities. Sources for nominations for 
members of the general public included civic organizations such as the League of Women Voters, the Bar 
Association, journalists, education editors for major newspapers, and informal networking. 

Each focus group consisted of ten to twelve members. Participants were required to be familiar with the 
knowledge and skills in either writing or civics that are typically possessed by students in at least one of 
the grades tested by NAEP. In addition, focus group members were required to be trained in the subject 
matter tested by NAEP or have related work experience in the area. 

Focus Group Sessions. Members received extensive advance materials to assist them in reviewing and 
imderstanding the ALDs before the meetings were convened. The sessions were lead by an experienced 
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focus group facilitator from Abt Associates. The meetings were held on Saturdays in November, 
December and January (1998-1999) and each lasted between three and four hours. 

The meeting format included a discussion of three main questions; 

1. Given the NAGB policy definitions of the three achievement levels (solid academic performance, 
partial nuistery, and superior performance) are the descriptions reasonable statements of what 
students should know and be able to do at each grade? 

2. Are the statements usefiil in describing student performance? That is, are they effective in 
communicating student performance? 

3. Given what you know about students, are the achievement level descriptions reasonable statements of 
what students should know and be able to do at each NAEP grade level? 

After a general group discussion of these questions, focus group members made specific 
recommendations that were intended to improve each description, at each grade, for each achievement 
level. The recommendations were summarized and reviewed by the entire group. Final corrections and 
revisions to the recommendations were discussed and recorded before the meeting was adjourned. 

This approach appeared to work well for the writing focus groups, but not as well for the civics groups. 
Because the preliminary ALDs for civics had been written as long hsts of very specific statements, civics 
group members had difficulty distinguishing these from test items. They were inclined to discuss the 
ALDs as if they were test questions. Members tended to make recommendations on how to restate 
specific descriptions rather than to make recommendations on how to combine and summarize the 
descriptions. After two sessions, ACT decided to change the task somewhat for the civics focus groups. 

The procedures previously outlined for facilitating the focus groups were followed for the last two 
meetings with one exception. In St. Louis and Portland, members of the civics focus groups were asked to 
recommend more general, broad, “umbrella-type” statements for the ALDs. To do this, members worked 
in small grade groups to generate the recommendations based on the following questions; 

• Is this description essential for solid performance? (Partial mastery? Superior performance?) 

• If this description were omitted, would the sense of solid performance be lost? (Partial mastery? 
Superior performance?) 

• Which elements can be combined to make the statements more useful? 

This change in procedures for the civics focus groups seemed to help members produce recommendations 
that were more relevant to the work of the Civics Expert Review Panel. Overall, the eight focus groups 
produced an abundance of rich, qualitative information about the preliminary ALDs. A complete set of 
recommendations by focus groups was compiled in each city. The following is a brief summary of the 
general recommendations made by the all of the focus groups. 

General Recommendations from the Writing Focus Groups: 

• The descriptions in general are too high for first draft writing. 

• Basic in particular is too high for 4* and 8* grades, but is about right for 12* grade. 

• Advanced for 12* grade should be higher. 

• The statements need to describe a clearer progression of writing skills across levels and grades. 

• The statements are too vague. They need to be more specific and use examples. 

• The statements need to use simpler, clearer language. 

General Recommendations from the Civics Focus Groups: 

• Basic is too high for all grade because it denotes more than “partial mastery.” 

• The content described in Basic is too difficult, especially at the 4* grade. 
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• Advanced is truly “superior performance” for 4* and 8* grades. 

• The statements need to describe a clearer progression of civics skill development across levels and 
grades. 

• The statements need to use simpler, clearer language. The language is inflated and most statements 
are too complex. 

• The descriptions for 4* grade should be based on concrete experiences that involve home, school, and 
community. The statements for 4* grade describe concepts that are too abstract to be developmentally 
appropriate. 

• The level of detail is inconsistent. In particular, the descriptions for 12* grade are very broad, whereas 
for 8* grade they are very specific. 

• There seems to be a gap between the performance described for 8* grade and 12 grade. 

Step 2: The Expert Review Panels (ERP) 

Within a few weeks of the last of the focus group meetings, the Expert Review Panels (ERP) for Civics 
and Writing met to review the recommendations fi'om the focus groups. The Expert Review Panels 
included members fi-om the NAEP Framework Panels, the NAEP Item Development Committees, and the 
NAEP Standing Committees for each content area. The Writing ERP consisted of five experts, three of 
whom attended the meeting, and the Civics ERP consisted of eight experts, seven of whom attended. All 
but one ERP member participated in a focus group meeting, and at least one ERP member was present at 
each focus group session. 

The Expert Review Panel Process. Members received advance materials that included all of the 
inf ormation utilized during the focus group sessions in addition to a summary report of the comments and 
recommendations generated by the focus groups. The ERP meetings were held on weekends and lasted 
for either two or three days. 

The ERP meeting format included the following procedures: 

1 . The ERP reviewed and discussed the preliminary ALDs and the recommendations offered by the 
focus groups. The review was thorough; every recommendation was considered for each ALD. 

2. After lengthy group discussion, the ERP reached preliminary agreement regarding how to revise, edit, 
modify, and reassign each statement. 

3. With the assistance of an editor, the ERP revised the statements and drafted a narrative, rather than 
bulleted version of the ALDs. 

4. Additional polishing of the narrative version of the descriptions took place following the ERP 
meeting. The polished version was distributed to the ERP for further consideration. After minor 
adjustments, members agreed to approve them as the recommended version of the ALDs. 

The ERP made every effort to strike a balance between implementing all of the recommendations offered 
by the focus groups, and maintaining a cohesive progression of skill development across the grades. In 
general, ERP members addressed the specific concerns expressed by the focus groups. In nearly all 
instances, the revisions followed the recommendations fi'om the focus groups. In a few cases, however, 
the panel agreed that the recommendation was inappropriate and was not implemented. Revisions used 
fewer jargon-type terms and created unambiguous statements. The group was attentive to providing 
adequate coverage for the subject area, as outlined in the Framework. The Civics ERP systematically 
addressed each content domain outlined in the Civics Framework. Members considered skill progression 
across all three grades to assure that the level was developmentally appropriate. The review process was 
thorough and the revisions were extensive. A complete set of modifications was compiled for each of the 
expert review panels. The following is a brief summary of the general changes made to the descriptions 
by the ERP. 
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General Modifications Made by the Expert Review Panels: 

• The degree of difficulty was adjusted as recommended. 

• The language was simplified. 

• Concepts were clarified. 

• Separate statements describing related ideas were consolidated. 

• The distinction between levels was enhanced. 

• The developmental progression of skills across levels and grades was improved. 

• The level of detail in the descriptions was made consistent across levels and grades. 

Step 3: Evaluating the Revised ALDs 

Once the recommended versions of the ALDs were finalized, ACT requested comments about the 
recommended ALDs fi’om the original focus group members and key people who had been involved in 
the development of the NAEP for writing and civics. 

Collecting Opinions from Focus Groups 

The original focus group members were asked to review the modified ALDs. Eighty-five participants 
were mailed copies of the revised ALDs and the generic NAGB policy definitions of Basic, Proficient, 
and Advanced. The focus group members were informed that a telephone interviewer would call them to 
ask their opinions of the recommended descriptions. Their opinion should be based on the reasonableness 
and usefulness of the revised descriptions, relative to the NAGB policy definitions. A toll-fi-ee telephone 
number was provided to members for returning calls to ACT. 

A copy of the telephone survey is presented as Exhibit 1 for civics and Exhibit 2 for writing. Results of 
the survey can be found in Tables 3 and 4 for civics focus group members, and in Tables 5 and 6 for 
writing focus group members. 

The results of the telephone survey indicate that the modified ALDs were generally well received by the 
persons who served on the focus groups. 

• When asked if the revised ALDs seem reasonable, 89.2% of the civics members and 84.6% of the 
writing members responded positively. 

• When asked if the revised ALDs are clear and easily imderstood, 78.4% of the civics members and 
79.5% of the writing members replied “yes.” 

• When asked if the revised ALDs reflect the three achievement levels, between 94.6% - 100% of 
civics members agreed (depending upon the level), and between 84.6% - 94.9 % writing members 
agreed. 

When asked if zidditional changes to the ALDs are necessary, 40.5% of the civics members and 48.7% of 
the writing members replied “yes.” Comments from these individuals consisted mainly of editorial type 
suggestions for the statements. Few made recommendations for changes that would likely lead to 
significant alterations of the levels. Although the percentages of participants who advocated additional 
changes seem rather high, 78.4% of civics members and 82.1% of writing members recommended that 
the ALDs be adopted as they appeared in the survey. 

The writing survey included one question that did not appear in the civics survey. This question (#7) 
asked participants to evaluate the descriptions for 8* grade Basic and Proficient, relative to one another. 
Specifically, members were asked whether there appears to be a “gap” in the progression of skill 
development described by the two levels. Of the 29 respondents to this question, 10 (34.5%) said that 
there was a “gap.” Basic seemed too low and Proficient too high for Grade Eight. 
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Reviews by Content Committees 

ACT requested comments about the recommended ALDs from key people who had been involved in the 
development of the Assessment for writing and civics. Members of the NAEP Standing Committee, the 
NAEP Item Development Committee, and the NAEP Framework Committee for each subject were asked 
to review the modified ALDs. Forty civics participants and 15 writing participants were contacted. Each 
was mailed copies of the recommended ALDs, the Framework, and the generic NAGB policy definitions 
of Basic, Proficient, and Advanced. The content committee members were asked to review the materials 
and make comments and suggestions for improvement to the recommended descriptions. A copy of the 
survey is presented as Exhibit 3 for civics and Exhibit 4 for writing. Results of the survey of civics 
content committee members are in Table 7 and of writing members in Table 8. 

Comments were received from 26 persons in civics and 7 in writing. The results indicate that the modified 
ALDs were generally well received by the persons who served on the content committees. 

• When asked if the revised ALDs should be adopted without substantive changes, 73 . 1 % of the civics 
members and 71.4% of the writing members said “yes.” 

• Of those who recommended changes, one or two members thought the descriptions were too high for 
some grades and levels, while one or two other members thought they were too low. 

• The greatest number of recommendations was made for the 4* grade Advanced level in civics. Four 
members thought that description was too demanding. 

Analyses comparing the Revised ALDs with the Framework and Item Pool 

The preliminary ALDs for both writing and civics have changed substantially as a result of the finalizing 
process. It is important to ascertain that the revised descriptions have not changed to such an extent that 
they no longer reflect the content described in the Assessment Framework. An analysis of the civics 
content areas as they relate to the revised civics ALDs revealed that all five areas of civic knowledge, as 
defined in the Framework were described by the revised ALDs. More specifically, the ALDs describe the 
civics content outlined in the Framework for all three grade levels, for all three achievement levels, and 
for a broad range of intellectual skills. 

Further analysis was done to classify the items in the civics exercise pool according to the revised 
achievement level descriptions. Sixty-five different descriptive statements were examined across the three 
grades. There were only two sentences - one at 4* grade Basic and the other at 8* grade Basic - that had 
no exercises associated with them. The findings of this analysis are summarized below: 

• The overall match between the civics item pool and the revised achievement level descriptors is 
strong. 

• Some descriptive statements can be associated with four or five assessment items, while others can be 
associated with twenty items. 

• A limited number of items from the exercise pool could not be directly associated with a descriptor. 

At each grade, over 90% of the exercise pools could be classified successfully according to the revised 
achievement level descriptions. The number of exercises that could not be classified was small. At grade 
four, 6 of the 90 items could not be classified as described by the ALDs. At grade eight, 10 of the 151 
items could not be classified and at grade twelve, 7 of the 152 items could not be classified. The majority 
of the unclassified exercises in grades 8 and 12 were designed to measure students’ abilities to interpret 
information presented in tables, graphs, or charts. This skill is included in the assessment specifications, 
but not in the ALDs. The other items that could not be classified measured students’ abilities in different 
areas that were not described in the ALDs. 
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Conclusion of the ALD Finalizing Process 

The 1998 achievement levels descriptions have received a great deal of attention, considerably more than 
has been given to ALDs in previous NAEP achievement levels-setting processes. The method of 
collecting opinions regarding the ALDs has been thorough and diligent. Great effort has been given to 
incorporating the collected opinions into the recommended ALDs. 

The ALDs have been further polished as a result of the continuing analyses comparing the revised ALDs 
with the item pool. Statements have been added to the 8* and 12*^ grade descriptions related to students’ 
abilities to interpret information presented in tables, graphs, and charts. Also, an explanation of testing 
conditions has been added as a preamble to the writing ALDs. 

At the NAGB meeting in August, 1998 the revised achievement levels descriptions were approved for 
ACT to use for the NAEP pilot studies and the ALS meetings. NAGB is scheduled to decide on adoption 
of the achievement levels for the 1998 NAEP in May 1999. 
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Composition of Civics Focus Group Participants 
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Civics Focus Group Follow-Up Survey Results by Location 
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Do the achievement level 
descriptions reflect solid academ 
performance at the Proficient 
level? 


□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low?) 


Do the achievement level 
descriptions reflect superior 
performance at the Advanced 
level? 

□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low?) 


Should the recommended 
achievement levels descriptions 
be adopted as they appear in the 
materials you received? 

□ Yes 

□ No 
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Writing Focus Group Follow-Up Survey Results by Grade and Type 
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Table 7 



Recommendations to Adopt Civics Achievement Levels Descriptions 



Recommend adoption of the civics ALDs without substantive changes 1 9 (73 , 1 %) 

Recommend substantive changes in some of the civics ALDs 7 (26,9%) 

Descriptions recommended for substantive changes 







Grade 4 


Grade 8 


Grade 


Too High 


Basic 


2 


0 


1 




Proficient 


0 


0 


1 




Advanced 


4 


1 


0 


Too Low 


Basic 


2 


• 0 


2 




Proficient 


0 


0 


2 




Advanced 


0 


0 


0 



Table 8 

Recommendations to Adopt Writing Achievement Levels Descriptions 



Recommend adoption of the civics ALDs without substantive changes 5 (7 1 ,4%) 

Recommend substantive changes in some of the civics ALDs 2 (28,6%) 

Descriptions recommended for substantive changes 







Grade 4 


Grade 8 


Grade 12 


Too High 


Basic 


0 


0 


1 




Proficient 


1 


1 


1 




Advanced 


1 


1 


1 


Too Low 


Basic 


0 


1 


0 




Proficient 


0 


0 


0 




Advanced 


0 


0 


0 
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Exhibit 1 



Telephone Survey for Civics Focus Groups 



(prefix) (fname) (Iname) (type) 

(address) 

(city), (state) (zip) 

(phone) 



A. Hello. My name is and I’m calling for ACT. The reason for my call is to ask your 

opinion about the recommended achievement level descriptions for the NAEP/National 
Assessment of Educational Progress in Civics. The interview should take about five 
minutes. Is this a good time to talk to you? 

If no. ask when to call back. 

B. Have you received a copy of the recommended achievement level descriptions? 

□ Yes (If yes, go on to next question.) 

□ No (If no, tell person we will send materials and ask for address where s/he wants 

materials sent. Tell person s/he will be called back for an interview soon after s/he 
has received materials.) 

C. Have you had a chance to review them? 

□ Yes (If yes, go on to next question.) 

□ No (If no, ask when to call back after s/he has had a chance to review materials. 



D. I’d like to ask you some questions about what you think about the descriptions. I’ll be 
taking notes while we talk, so there could be some pauses during our conversation while I 
write down what you’re saying. 

My first question is; 

1 . Do the achievement level descriptions seem reasonable? 

□ Yes 

□ No (If no, please explain.) 



2. Are the achievement level descriptions clear and easily understood? 

□ Yes 

□ No (If no, please explain.) 



3. Are additional changes to the achievement level descriptions necessary? 
□ Yes (If yes, please explain.) 



□ No 

4. Do the achievement level descriptions reflect partial mastery at the Basic level? 

□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low? 

5. Do the achievement level descriptions reflect solid academic performance at the Proficient 
level? 

□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low?) 

6. Do the achievement level descriptions reflect superior performance at the Advanced level? 

□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low?) 

And the final question, 

7. Should the recommended achievement levels descriptions be adopted as they appear in the 
materials you received (title of materials: Recommended Achievement Levels Descriptions 
for Civics)! 

□ Yes 

□ No 

Thank you for your help. 




19 



30 



Exhibit 2 



Telephone Survey for Writing Focus Groups 



(prefix) (fname) (Iname) (type) 

(address) 

(city), (state) (zip) 

(phone) 



A. Hello. My name is and I’m calling for ACT. The reason for my call is to ask your 

opinion about the recommended achievement level descriptions for the NAEP/National 
Assessment of Educational Progress in Writing. The interview should take about five 
minutes. Is this a good time to talk to you? 

If no. ask when to call back. 

B. Have you received a copy of the recommended achievement level descriptions? 

□ Yes (If yes, go on to next question.) 

□ No (If no, tell person we will send materials and ask for address where s/he wants 

materials sent. Tell person s/he will be called back for an interview soon after s/he 
has received materials.) 

C. Have you had a chance to review them? 

□ Yes (If yes, go on to next question.) 

□ No (If no, ask when to call back after s/he has had a chance to review materials. 



D. I’d like to ask you some questions about what you think about the descriptions. I’ll be 
taking notes while we talk, so there could be some pauses during our conversation while I 
write down what you’re saying. 

My first question is: 

1 . Do the achievement level descriptions seem reasonable? 

□ Yes 

□ No (If no, please explain.) 



2. Are the achievement level descriptions clear and easily understood? 

□ Yes 

□ No (If no, please explain.) 
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3. Are additional changes to the achievement level descriptions necessary? 
□ Yes (If yes, please explain.) 



□ No 

4. Do the achievement level descriptions reflect partial mastery at the Basic level? 

□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low? 

5. Do the achievement level descriptions reflect solid academic performance at the Proficient 
level? 

□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low?) 

6. Do the achievement level descriptions reflect superior performance at the Advanced level? 

□ Yes 

□ No (If no, are they: 

□ Too High? 

□ Too Low?) 

7. You don’t have to answer this next question if you feel uncomfortable making the judgment 
about 8**’ grade descriptions. But when you were looking over the descriptions for 8 grade, 
did you notice a gap between Basic and Proficient? More specifically, did Basic seem too 
low and Proficient too high? 

□ Yes 

□ No 

And the final question, 

8. Should the recommended achievement levels descriptions be adopted as they appear in the 
materials you received (title of materials: Recommended Achievement Levels Descriptions 
for Writing)! 

□ Yes 

□ No 

Thank you for your help. 
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Exhibit 3 



NAEP Civics Content Committee Survey 



March 25, 1998 

(prefix) (fname) (Iname) (type) 

(title) 

(organization) 

(address) 

(city), (state) (zip) 

Dear (prefix) (Iname); 

In preparing our report to NAGB, we have determined that we need specific responses from key people, 
such as yourself, who have been involved in the development of the NAEP for Civics. NAGB is sure to 
want your input on this matter! 

Please take a few moments to review the Recommended Achievement Levels Descriptions for Civics and 
the NAGB policy definitions for Basic, Proficient, and Advanced performance. We have enclosed a copy 
of each, so you do not need to tax your memory or search aroimd your office to find them. 

Please check the appropriate response to indicate your opinion about the reasonableness and usefiilness of 
the recommended achievement levels descriptions and how well they reflect the NAGB policy 
definitions. 

□ I recommend adoption of the Recommended Achievement Levels Descriptions for Civics. In my 
opinion, no substantive changes need to made to the descriptions. [Please skip to the instructions 
below.] 

□ I recommend substantive changes in some of the descriptions. These changes are necessary before I 
could recommend adoption of the Recommended Achievement Levels Descriptions for Civics. 

[Please respond to the following questions.] 

If you think changes are necessary, please indicate which description(s) you think should be modified and 
the general direction of change needed. ‘Too high” means that you think that description is too 
demanding and that it would result in a standard that is “too high” to be a reasonable reflection of the 
level. ‘Too low” means that you think the description is not demanding enough and that it would result 
in a standard that is “too low” to be a reasonable reflection of the level. 



Grade 4 


Basic 


Proficient 


Advanced 


Acceptable 


(Partial Mastery) 
□ 


(Solid Performance) 
□ 


(Superior Performance) 
□ 


Too high 


□ 


□ 


□ 


Too low 


□ 


□ 


□ 


Grade 8 


Acceptable 


□ 


□ 


□ 


Too high 


□ 


□ 


□ 


Too low 


□ 


□ 


□ 



ERIC 
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Grade 12 

Acceptable □ □ □ 

Too high □ □ □ 

Too low □ □ □ 



PLEASE RETURN THE SURVEY AT YOUR EARLIEST CONVENIENCE. OR BY APRIL 6. 



Your comments on specific recommendations will be appreciated. If you have already sent your 
recommendations, you do not need to repeat them here. If you wish to telephone, fex or E-mail your 
comments, please do so. A self-addressed, prepaid envelope is enclosed for you to return this form by 
mail. 

Telephone: (800) 525-6930 Fax; 319/337-1497 E-mail; hanick@act.org 
Thank you in advance for your continued assistance and support for the Civics NAEP! 



Yours truly. 




NAEP Project Director 
Research Division 
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Exhibit 4 



NAEP Writing Content Committee Survey 



March 25, 1998 



(prefix) (fname) (Iname) (type) 

(title) 

(organization) 

(address) 

(city), (state) (zip) 

Dear (prefix) (Iname); 

In preparing our report to NAGB, we have determined that we need specific responses from key people, 
such as yourself, who have been involved in the development of the NAEP for Writing. NAGB is sure to 
want your input on this matter! 

Please take a few moments to review the Recommended Achievement Levels Descriptions for Writing and 
the NAGB policy definitions for Basic, Proficient, and Advanced performance. We have enclosed a copy 
of each, so you do not need to tax your memory or search around your office to find them. 

Please check the appropriate response to indicate your opinion about the reasonableness and usefulness of 
the recommended achievement levels descriptions and how well they reflect the NAGB policy 
definitions. 

□ I recommend adoption of the Recommended Achievement Levels Descriptions for Writing. In my 
opinion, no substantive changes need to made to the descriptions. [Please skip to the instructions 
below.] 

□ I recommend substantive changes in some of the descriptions. These changes are necessary before I 
could recommend adoption of the Recommended Achievement Levels Descriptions for Writing. 
[Please respond to the following questions.] 

If you think changes are necessary, please indicate which description(s) you think should be modified and 
the general direction of change needed. ‘Too high” means that you think that description is too 
demanding and that it would result in a standard that is “too high” to be a reasonable reflection of the 
level. ‘Too low” means that you think the description is not demanding enough and that it would result 
in a standard that is “too low” to be a reasonable reflection of the level. 



Grade 4 


Basic 


Proficient 


Advanced 


Acceptable 


(Partial Mastery) 
□ 


(Solid Performance) 
□ 


(Superior Performance) 
□ 


Too high 


□ 


□ 


□ 


Too low 


□ 


□ 


□ 


Grade 8 


Acceptable 


□ 


□ 


□ 


Too high 


□ 


□ 


□ 


Too low 


□ 


□ 


□ 
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□ 

□ 

□ 



□ 

□ 

□ 



□ 

□ 

□ 



Grade 12 
Acceptable 
Too nigh 
Too low 



PLEASE RETURN THE SURVEY AT YOUR EARLIEST CONVENIENCE. OR BY APRIL 6. 



Your comments on specific recommendations will be appreciated. If you have already sent your 
recommendations, you do not need to repeat them here. If you wish to telephone, fax or E-mail your 
comments, please do so. A self-addressed, prepaid envelope is enclosed for you to return this form by 
mail. 

Telephone: (800) 525-6930 Fax: 319/337-1497 E-mail: hanick@act.org 

Thank you in advance for your continued assistance and support for the Writing NAEP! 



Yours truly. 




NAEP Project Director 
Research Division 
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